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ABSTRACT 



This study focused on response rates to online student 



ratings of faculty at Brigham Young University, Utah, where concerns about 
response rates have contributed to a long period for testing and 
implementation of the online system (more than 5 years) . The first pilot 
study, in 1997, included 36 courses and yielded a response rate of about 40%. 
The second pilot, in 1999, included 194 course sections and ratings from 
8,285 students, with a response rate of 51%. As part of this pilot study, 
online and paper ratings were compared for 74 course sections. The online 
response rate for this subset of courses was 50%, while the paper response 
rate was 71%. The third pilot, in fall 2000 included 47 course sections and 
3,076 students, with a response rate of 62%. The fourth pilot study was 
planned for winter 2002. Findings from the three initial studies do not 
indicate that length of the form was a factor in low response rate. Low 
response rates were not negatively biased; students who were dissatisfied 
were no more likely than others to respond online. Results of the studies 
support the findings of previous research that students are more likely to 
respond if they believe the ratings will be used for decisions about courses 
and faculty. As current response rates approach 70%, and research shows ways 
to increase response rates, online student ratings are likely to succeed at 
Brigham Young University. (SLD) 
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Online Student Ratings: Will Students Respond? 



Background 

Student ratings of instruction are the most common means of evaluating teaching at US 
colleges (Seldin, 1999). Student ratings are administered almost exclusively in class using paper- 
pencil questionnaires — a costly, time-consuming process that is inconvenient for faculty and 
often restricts the thoughtfulness and depth of student responses. 

The increasing use of technology in education, especially the World Wide Web, has led 
to the possibility of online administration and reporting of student ratings. In a recent survey of 
the 200 most wired colleges in the United States, 25% of respondents said they were already 
using or plan to covert to online student ratings (Hmieleski, 2000). A Web search (February, 
2002) of institutions using online evaluations yielded over 80 universities using online student 
ratings for some courses (e.g., specific colleges or departments, online courses) and three 
universities using online ratings for all courses on campus. 

Online student ratings have many potential benefits, including reduced processing time 
and costs; more accurate data collection and reporting; more detailed, user-friendly reports; ease 
of administration; more complete data collection; and longer, more thoughtful student responses. 
There are also some major obstacles to successful implementation of online ratings, most notably 
obtaining adequate response rates. 

Response Rates for Online Ratings 

Unlike traditional paper-pencil student ratings that are administered in class, online 
ratings are completed outside of class during students’ discretionary time. This allows students 
more time to fill out the rating forms, but it also gives them more freedom in their decision of 
whether or not they complete the forms. 

Many institutions have identified response rate as a challenge to online ratings (e.g., 

Duke University, Georgia Institute of Technology, Air Force Academy, Kansas State University, 
Northwestern University, University of Idaho). 1 Universities using online ratings have 
implemented a number of strategies to increase response rates. Many universities ask faculty 
members to encourage students to complete the forms. Another common strategy is to help 
students understand the importance of their input and how student-rating results are used. In 
addition, Polytechnic University (New York) enters participating students in a drawing for palm 
pilots and course grades do not appear on students’ transcripts until they complete course ratings. 
Murdoch University (Australia) sends students multiple email reminders and enters participants 
in a drawing for a cash prize. Northwestern University provides student-rating results to students. 
Georgia Tech provides detailed information to students and faculty on the online rating system 
and its use. 



1 This information was obtained from university websites that address campus online student ratings (February, 
2002). 
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Studies of Response Rates at Brigham Young University 

Obtaining adequate response rates has been a primary concern in considering online 
student ratings at Brigham Young University (BYU). Concerns about response rates have 
contributed to a particularly long period for testing and implementation of the online system 
(over five years). These concerns have also spawned a number of studies to determine expected 
online response rates, analyze response rates under different conditions, and determine how 
response rates can be improved. 

The first pilot of online student ratings at BYU was conducted during Winter Semester 
1997. This pilot included 36 courses and yielded a response rate of approximately 40%. The 
second pilot, conducted in 1999, included 194 course sections and 8,285 students. This pilot 
yielded a response rate of 51 %. As part of the second pilot, online and paper rating forms were 
compared in 74 course sections. For these 74 sections, the online ratings response rate was 50% 
for online ratings compared to 71% for the paper-pencil ratings. The third pilot was conducted in 
Fall 2000 and included 47 course sections and 3,076 students. The response rate was 62%. A 
fourth and final pilot of online student ratings is being conducted Winter 2002. 

Possible Reasons for Increase in Response Rates Over Time 

Data collected from faculty and students at BYU suggest a number of factors that may 
increase response rates for online student ratings. These factors include: 1) Student access to 
computers, 2) amount and quality of communication to faculty and students regarding the online 
rating system, 3) communication to students regarding how student ratings are used, and 4) 
faculty and student support of the online rating system. These factors have generally increased 
over the time in which the pilots were conducted at BYU. 

Student access to computers has steadily increased. Most BYU students have their own 
computers. Some colleges on campus have recently required all students in the college to have 
laptop computers. The number of computers in computer labs has also increased. 

Communication to faculty and students became more frequent with each pilot. Steps were 
taken to increase the clarity of the communication. Also there is evidence that in the later pilot 
studies, more faculty and students actually received and read the messages that were sent via 
email. 

Some recent efforts have been made to help students understand how rating results are 
used. But this information has not been distributed widely. More work is needed in this area. 

There are some indications that faculty and students have become more supportive of 
online student ratings at BYU. Recent student surveys and focus groups have yielded high levels 
of support for online ratings. Faculty support is more guarded but is also increasing. In addition, 
the communication from faculty members to students about online ratings increased over time. 
More faculty members encouraged or assigned students to complete the forms in the later pilot 
studies. 

Response Rates Under Various Conditions 

Faculty members participating in the third pilot were asked to report on their 
communication to students regarding completion of the online rating forms. Seventeen (50%) of 
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the 34 participating faculty members reported what they said to students. Their communication 
to students is categorized as follows: 

• Instructors assigned students to complete the online student-rating form and gave 
students points for completing the assignment (7 instructors) 

• Instructors assigned students to complete the online student-rating form but did not 
give points for completing the assignment (4 instructors) 

• Instructors encouraged students to complete the online rating form but did not make it 
a formal assignment (4 instructors) 

• Instructors did not mention the online student-rating form to students (2 instructors) 

The following table shows the categories of faculty communication to students with the 
student-rating response rates in each category. 

Response Rate by Communication Category 



Communication Response rate 



Assigned students to complete online rating forms and gave them points 
for doing so 


87% 2 

(range: 59%-95%) 


Assigned students to complete online rating forms but did not give them 
points 


77% 

(range: 31%-100%) 


Encouraged students to complete the online forms but did not make it a 
formal assignment 


32% 

(range: 17%-41%) 


Did not mention the online student-rating forms to students 


20% 

(range: 8%-42%) 



When faculty members assigned students to complete the online rating forms (whether or 
not points for the assignment were given), response rates greatly increased. It is not clear why 
there was such a wide range of response rates in each communication category. 

Response Rate and Length of Student Rating Forms 

Three different forms (long, 18 items; medium, 10 items; short, 6 items) were used in the 
third pilot study. Different forms were used to determine the amount of time students took to 
complete different lengths of forms and to see if the length of form made a difference in response 
rate. 

The average time it took to complete the long form was 3 minutes 5 seconds. 3 The 
average time to complete the medium form was 2 minutes 18 seconds. The average time to 
complete the short form was 2 minutes 29 seconds. The average time it took students to complete 
forms for all their courses was 1 3 minutes 47 seconds. 4 



2 Average response rate across classes in this category, not weighted according to number of students in each class. 

Responses taking more than V 2 hour were removed when calculating means for completing individual forms 
because these few responses were clear outliers and likely occurred when students did things other than completing 
the form during the time the rating screen was open. 

Responses taking more than 1 hour were removed when calculating the mean for completing forms for all courses 
because these few responses were clear outliers and likely occurred when students did things other than completing 
the forms during this time period. 
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Nearly all students in the focus groups agreed that the length of the rating forms was not 
a factor in their decisions to complete the forms (i.e., the forms were quick to complete online 
regardless of the different lengths of the forms). 

Students Completing Rating Forms for All Their Courses 

Some faculty members and administrators at BYU assumed that if students accessed the 
online system to rate one of their courses, they would go ahead and rate all their courses. This 
was not necessarily the case. In the third pilot, data were collected on the number of students 
who completed rating forms for all their courses. Of the 1,892 pilot students who completed 
forms for at least one of their courses, 638 (34%) completed forms for all their courses. 

Even though there were a low percentage of students rating all their courses, pilot results 
showed that students who had more than one course included in the pilot study were more likely 
to rate all their courses. Of the 62 responding students who were in two pilot courses, 25 (40%) 
completed forms for all their courses. Of the 38 responding students who were in three pilot 
courses 28 (74%) completed forms for all their courses. It appears that when completion of 
online rating forms is assigned or encouraged in more than one course (as was the case in most 
of the pilot courses), the likelihood of responding students completing forms for all their courses 
improves considerably. 

Response Rate for Open-Ended Comments 

Of the 17,279 rating forms completed in the third pilot, 10,943 (63%) included students’ 
written comments. This is much more than the number of forms containing written comments 
using the current in-class, paper rating system (i.e., less than 10% now include written 
comments). In addition, the length of the written comments for online rating forms was greater 
than the length of comments using the current system. 

Students completing the short form were more likely to make written comments. On the 
short form, 71% of responding students made written comments compared to 63% on the long 
form and 61% on the medium fomi. 

Why Some Students Didn’t Respond 

In the telephone survey of non-respondents (i.e., those in the pilot study who did not 
complete online rating forms), students were asked why they did not complete the online rating 
forms. Twenty-five students were contacted. Of these 25 students, 1 1 said they were aware that 
one or more of their courses was included in the pilot study. Of these 1 1 students, 4 said they did 
complete the online rating forms, 6 said they did not complete the forms, and 1 could not 
remember if s/he completed the forms or not. The 6 students were asked why they did not 
complete the forms. Four of the 6 students said they forgot. The remaining 2 students said they 
didn’t complete the online forms because they were required to complete paper-pencil forms in 
class and the online forms were optional. It is unclear why 14 of the 25 students were not aware 
of their participation in the pilot study. 
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Possible Bias When Response Rates are Low 



Both faculty and students at BYU have expressed concern that low response rates may 
cause a negative bias in student rating results. The hypothesis is that students who are upset or 
disappointed in a course or instructor are more likely to complete online rating forms. Therefore, 
a low response rate would yield a negative bias in overall ratings. To test this hypothesis, both 
online and paper-pencil rating forms were administered in 74 course sections. 

Web-Paper Correlation . The correlations between the online and the paper forms were 
0.84 (overall instructor) and 0.86 (overall course). On the average, the overall course and 
instructor ratings were 0.1 points higher for the online ratings. On the online form, 67.6% of the 
sections had the same or higher overall instructor ratings. The remaining 33% had overall 
instructor ratings within 0.1 to 0.5 points of the paper rating. There was no evidence that low 
response rates on the online ratings resulted in consistently lower rating results. 

Rating-Response Rate Correlation . For the paper form, 8% of the sections had less than 
40% response rates. The minimum response rate was 13%. The correlation between the response 
rate and the overall ratings was 0.41. This correlation accounted for about 17% of the variance in 
ratings. It appears that the higher the percent attendance on the day the paper form was handed 
out, the better the overall ratings. 

For the online form, 18% of the sections had less than a 40% response rate. The 
minimum response rate was 33%. The correlation between response rate and overall ratings was 
0.09. If a low web response rate biased the results, there would have been a much higher 
correlation between ratings and response rate. 

Strategies to Increase Response Rates 

Faculty members responding to a survey identified low response rates as an obstacle to 
successful online ratings but said little about how to achieve adequate response rates. One faculty 
member suggested that completion of the online forms should be a homework assignment. 
Another faculty member was concerned that students might resent having to complete the rating 
forms as assignments. 

Students participating in the third pilot expressed concerns about achieving adequate 
response rates using online ratings. Students also provided a number of suggestions to increase 
response rates: 5 

• Withhold students’ early access to grades until they logon to the online rating 
system. This would be effective yet not too restrictive (nearly all students in the 
focus groups supported this approach) 

• Provide extra credit/points for those who complete the forms 

• Provide education and instructions regarding the online rating system 

• Encourage instructors to show a personal interest in students completing the 
forms (e.g., instructors could mention the forms in class, let students know that 
they pay attention to student responses, or send personal emails to students 
reminding them to complete the forms) 



5 Student comments are listed in order 



of frequency with the most frequent comments first 
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• Provide positive incentives (e.g., contribute money to a charity for each form 
completed, give students coupons for food or for the Bookstore for completing the 
forms) 

• Provide greater student access to computers 

Discussion 

Findings from the BYU studies help answer three primary questions about response rates 
and online student ratings: What contributes to low response rates for online ratings? Do low 
response rates bias results? How can response rates for online ratings be increased? 

What contributes to low response rates for online ratings ? Research was conducted to see 
if the length of the online student rating form affected response rates. Forms containing 18, 10, 
and 6 items were tested. The length of the form did not appear to be an important factor in 
students’ decisions to complete the forms, although there would undoubtedly be a threshold at 
some point. Eighteen items was not too many, but how many items are too many? At what point 
would response rates be significantly influenced? What factors affect the response rate in relation 
to the number of items on the rating form? In the future, faculty, chairs, and deans will be able to 
add items to the BYU online rating form. Research on response rates when items are added to the 
form may help answer these questions. 

Do low response rates bias results ? It is interesting that response rates for online ratings 
weren’t negatively biased, even when response rates were relatively low. Apparently, students 
who are dissatisfied with a course are not more likely than other students to respond to online 
ratings. Or at least, dissatisfied students are not more likely to respond than students who are 
happy with the course (i.e., perhaps only the very positive and very negative students could be 
responding, resulting in a mean similar to that of the whole class). 

Further research is needed on levels of response rates and online ratings. Can the results 
of the BYU study be replicated and generalized? If low online response rates have little affect on 
course rating means, can low response rates be accepted as valid? What might explain the lack of 
correlation between response rates and online rating results, especially compared to paper-pencil 
ratings where there is a significant correlation between response rates and course student-rating 
means? Answers to these questions may help determine how, and to what extent, online student 
ratings are used in the future. 

How can response rates for online student ratings be increased ? Research on student 
ratings (Ballantyne, 1999) suggests that an important factor in response rates is students’ belief 
that rating results are used for important decisions about courses and faculty. Results of the BYU 
studies support this finding. Many students commented on the importance of faculty members 
using the student-rating results. Some were adamant, almost demanding, in their insistence that 
faculty members see and pay attention to what they said in the open-ended comments. Students 
wanted to be heard and know that their responses can make a difference. If students believed this 
was the case, many were motivated to complete the online rating forms. 

It was surprising how many students supported withholding early access to grades for 
those who had not completed the online student rating forms (or withholding early access to 
grades to those who had not at least logged on to the online rating system). Students thought this 
strategy would be effective, yet not too restrictive. 

Faculty members assigning students to complete the online rating forms had a 
considerable impact on response rates. This was true whether or not points were given for the 
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assignment, albeit response rates were generally the highest when points were given. Why was 
merely assigning so effective, even in cases where points were not given? One might 
hypothesize that students thought their participation in online ratings might affect their standing 
in the course regardless of whether or not points were actually given for the assignment. Or, the 
impact of instructors assigning students to complete the forms may have come from the message 
this sent to students that completing the forms was important to the faculty member. (If 
instructors care enough to make it an assignment, maybe they also pay attention to and use the 
results.) 

Conclusion 

At BYU, response rates are the primary obstacle to the use of online student ratings of 
instruction. Research has revealed a number of strategies that may be effective in increasing 
response rates, including: student access to computers, communication to faculty and students 
regarding the online rating system, communication to students about uses of rating results, and 
faculty and student support of the online rating system. Efforts to address these areas have 
generally resulted in increased response rates. Current response rates are approaching 70%. If 
response rates consistently reach this level, online student ratings are likely to succeed at BYU. 
Strategies used to increase response rates at BYU may be useful to other institutions that are 
using or considering online student ratings. 
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